Skip to content

feat: add binary title ID and save ID extraction during scan - #3925

Open
tmgast wants to merge 14 commits into
rommapp:masterfrom
tmgast:feature/sigil-title-ids
Open

feat: add binary title ID and save ID extraction during scan#3925
tmgast wants to merge 14 commits into
rommapp:masterfrom
tmgast:feature/sigil-title-ids

Conversation

@tmgast

@tmgast tmgast commented Jul 23, 2026

Copy link
Copy Markdown
Member

Description

RomM derives a game's identity from its filename during a scan, which is fragile in the cases that matter most. Base, update, and DLC files that do not carry a title ID in their name go uncategorized, Switch retail titles cannot be distinguished without decryption, and save-sync clients have no reliable way to pair a save to the folder an emulator actually creates on disk. This reads the platform-native identifier directly out of the ROM binary during the scan's file pass, using argosy-sigil (a native helper library, MPL-2.0), and stores it as typed columns surfaced in the v2 game details.

Extraction covers PSP, PS1, PS2, PS Vita, Switch, Switch 2, 3DS, Wii, and GameCube, and runs independently of the hash-database gate so it reaches non-hashable platforms like Switch. sigil is an optional dependency: the backend imports it behind a guard, so an image or dev environment built without it skips extraction rather than failing. Switch decryption uses the user's prod.keys from the platform's firmware folder (already scanned before roms); without it, encrypted retail titles degrade gracefully and decrypted dumps still resolve keyless.

For Switch, the NCA program ID alone reports the base title for every content type, so an update is indistinguishable from its base and no version is available. sigil instead reads the content metadata (CNMT), which yields each file's authoritative title ID, content type, and version. Base, update, and DLC are then categorized from the binary content type rather than folder names, an update resolves to its real ...800 id with its version, and DLC to its own id.

Changes

  • Per-file identity. Every applicable file gains title_id and save_id; the rom gains a base title_id, save_id, and save_usage. save_id is the on-disk save identity an emulator creates, so clients building save-sync pair saves by folder rather than by guessing from filenames.
  • Switch categorization from the binary. RomFileCategory is set from the CNMT content type (application/patch/addon to GAME/UPDATE/DLC), authoritative over folder names when extraction succeeds and falling back to the existing folder logic otherwise. The rom-level base id is derived from the per-file ids.
  • PS2 folder-prefix saves. A PS2 game can create several memory-card folders sharing a serial prefix (Champions of Norrath splits into BASLUS-20642SYS and BASLUS-20642RD0), so save_id is the region-prefixed stem and save_usage is folder-prefix; consumers enumerate every folder that starts with it.
  • 3DS split saves. The 3DS nests a save under two directory levels (title/<high 8>/<low 8>/), so save_id is the /-separated path (00040000/00033500) and save_usage is folder-split; consumers create the nested folders instead of re-deriving the split from the id.
  • Opt-in Switch filename embedding. A new config.yml flag filesystem.embed_switch_title_ids (default off) renames Switch files during a scan to embed the binary id and version (Name [TITLEID][vN].ext) and reconciles the database record. It is idempotent (skips names that already carry a title-id bracket), collision-safe, and Switch-only. This is the binary-driven form of what filename-parsing tools consume.
  • Packaging. sigil builds in a dedicated Docker stage (the RAHasher pattern), pinned by commit, and is copied into the venv. The guarded import keeps it optional everywhere else.
  • Frontend. Regenerated API types plus the per-file Title ID and Version chips on the game details Files tab; no other hand-written frontend changes.

Tests

  • Adapter: content-type and version mapping, absent-binding degradation, per-platform gating.
  • Filesystem handler: nested update and DLC extraction, content-type-over-folder category precedence, title_version capture, rom-level base selection and derivation, and 3DS folder-split save_usage mapping.
  • Embedding: rename format, idempotent skip, collision skip, non-Switch never renamed, and single-file (Rom.fs_name) versus multi-part (inner RomFile) reconciliation, including that title_version survives the file reconciliation.
  • sigil: synthetic in-memory NCA, CNMT, and 3DS (NCSD) unit tests, plus validation against real retail Switch (Moonlighter, Dragon Quest Builders 2), PS2, and 3DS (Dillon's Dead-Heat Breakers) dumps.
  • Migrations 0104, 0105, and 0106 (the latter adds folder-split to the saveusage enum) upgrade and downgrade on MariaDB and PostgreSQL.
  • Verified end-to-end against real Switch, PS2, and 3DS dumps, including per-file categorization, the 3DS nested save_id, and the embedding rename with database reconciliation.

Known Gaps

  • Distinguishing a Switch base from its update requires the CNMT, so loose base and update files in one folder without keys cannot be told apart; the folder layout and keyed extraction cover the supported cases.
  • Archived-content members are out of scope; extraction runs on served files, not entries inside archives.

AI Disclosure
Planning and implementation supported by Claude Code.

Closes #2526

Checklist

  • I've tested the changes locally
  • I've updated relevant comments
  • I've assigned reviewers for this PR
  • I've added unit tests that cover the changes

tmgast added 11 commits July 24, 2026 02:54
New SigilService adapter wraps the optional sigil cffi binding (guarded
import, degrades to a no-op when absent) and extracts platform-native
title/save IDs from ROM binaries during the scan's file pass, gated to
nine platforms and independent of the hashable-platform gate. Switch
resolves prod.keys from the platform's scanned firmware. Per-file
title_id/save_id land on rom_files, rom-level title_id/save_id and a
SaveUsage enum land on roms (base Switch ID derived arithmetically when
only update/DLC files exist). Config flag
filesystem.skip_title_id_extraction mirrors skip_hash_calculation.
Regenerated API types (title_id/save_id on RomFileSchema, plus
save_usage and the SaveUsage enum on rom schemas), Title ID / Save ID
rows on the metadata tab, a click-to-copy Title ID chip per file row,
and the new rom.title-id / rom.save-id locale keys across all locales.
docker/Dockerfile gains a sigil-build stage (python:3.13-alpine base so
the cffi extension matches the venv's cp313/musl ABI) whose staged
package is copied into the slim image's site-packages; the Ubuntu dev
image builds it inline after uv sync. Pinned by commit ARG. The
backend treats the package as optional, so images built without it
simply skip extraction. DEVELOPER_SETUP documents the optional local
build.
Drop the top-level-only gate so update and DLC files in nested folders
get extracted. Consume sigil's new content type and version: set the
RomFile category from the binary content type (application/patch/addon
-> GAME/UPDATE/DLC), authoritative over folder names for Switch when
CNMT succeeds, and store the title version in a new BigInteger column.
Updates now carry their own ...800 id and version instead of collapsing
to the base.
New config.yml flag filesystem.embed_switch_title_ids (default off)
renames Switch/Switch 2 files during scan to embed their binary
title id and version (Name [TITLEID][vN].ext), then reconciles the DB
record. Idempotent (skips names already carrying a title-id bracket),
collision-safe, and Switch-only. Single-file roms update Rom.fs_name;
multi-part inner files update only their RomFile name. Driven by the
CNMT-extracted per-file ids, this is the correct form of the filename
embedding the original PR group attempted via filename heuristics.
Adds a compact version chip beside the per-file Title ID, shown only
when the version is meaningful (hidden for the 0 that base games and
DLC report), with the rom.version locale key across all locales.
The purge-and-recreate list in _identify_rom copied title_id and
save_id but not the new title_version column, so extracted Switch
update versions were dropped to NULL on persist. Copy title_version
too, and add a regression test that captures the rebuilt RomFile and
fails if the version is lost.
Re-integrate the sigil title-id feature on top of upstream's scan
reconciliation refactor and migration additions.

Scan reconciliation:
- _identify_rom now persists rom files via db_rom_handler.sync_rom_files
  instead of purge_rom_files + per-file add_rom_file. The obsolete
  RomFile copy-list (title_id/save_id/title_version) is removed; those
  fields now ride along through ROM_FILE_SCANNED_COLUMNS.
- Add title_id, save_id, title_version to ROM_FILE_SCANNED_COLUMNS so
  sync_rom_files copies them onto reconciled rows. Without this the
  extracted ids/version silently drop to NULL on rescan.
- Single-file embedding rename (renamed_rom_fs_name) preserved in both
  the new-entry and update paths of _identify_rom.

Migrations renumbered to chain after the new upstream head:
- 0102_sigil_title_ids -> 0104_sigil_title_ids (down: 0103_roms_facets_provider_ids)
- 0103_sigil_title_version -> 0105_sigil_title_version (down: 0104_sigil_title_ids)

Tests:
- Reworked TestIdentifyRomPersistsFileTitleVersion to capture
  sync_rom_files and assert the scanned RomFile carries title_id and
  title_version; added a direct assertion that ROM_FILE_SCANNED_COLUMNS
  includes the sigil columns.
@tmgast tmgast closed this Jul 23, 2026
@tmgast tmgast reopened this Jul 23, 2026
@tmgast tmgast changed the title feat(scan): binary title ID and save ID extraction via sigil Add binary title ID and save ID extraction during scan Jul 23, 2026
@tmgast
tmgast marked this pull request as ready for review July 23, 2026 23:11
Copilot AI review requested due to automatic review settings July 23, 2026 23:11
@greptile-apps

greptile-apps Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds native ROM identity extraction and optional Switch filename embedding.

  • Stores per-ROM and per-file title IDs, save IDs, save usage, and Switch title versions.
  • Uses extracted Switch content metadata to categorize base games, updates, and DLC.
  • Adds schema migrations, guarded sigil integration, Docker packaging, generated API types, UI chips, localization, and tests.

Confidence Score: 3/5

The PR should not merge until existing libraries receive a reliable identity-field backfill; the non-atomic filename rename should also be hardened.

Default, scheduled, and watcher-driven scans can leave all newly added identity columns null for pre-existing ROMs, while embedding changes filenames before the database update is secured.

backend/endpoints/sockets/scan.py, backend/handler/filesystem/roms_handler.py, backend/alembic/versions/0104_sigil_title_ids.py, backend/alembic/versions/0105_sigil_title_version.py

Important Files Changed

Filename Overview
backend/endpoints/sockets/scan.py Integrates extraction into the scan pipeline, but ordinary scans do not populate the new fields for existing libraries.
backend/handler/filesystem/roms_handler.py Implements extraction, Switch categorization, base-ID derivation, and embedding; embedding mutates files before database reconciliation.
backend/adapters/services/sigil.py Provides a guarded asynchronous adapter that degrades cleanly when the optional native binding or identifiers are unavailable.
backend/models/rom.py Adds nullable persisted identity fields and the typed save-usage enum.
backend/alembic/versions/0104_sigil_title_ids.py Adds identity columns and indexes but intentionally or inadvertently leaves existing rows without extracted values.
backend/alembic/versions/0105_sigil_title_version.py Adds the nullable per-file title-version column without arranging an existing-library backfill.
docker/Dockerfile Builds the pinned sigil native dependency in a dedicated stage and copies its Python binding into the runtime environment.
frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue Displays the newly exposed per-file title ID and version metadata.

Comments Outside Diff (1)

  1. backend/endpoints/sockets/scan.py, line 231-236 (link)

    P1 Existing ROM identities stay empty

    When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents get_rom_files from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.

    Knowledge Base Used: ROM Scanning Flow

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: backend/endpoints/sockets/scan.py
    Line: 231-236
    
    Comment:
    **Existing ROM identities stay empty**
    
    When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents `get_rom_files` from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.
    
    **Knowledge Base Used:** [ROM Scanning Flow](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/rom-scanning-flow.md)
    
    How can I resolve this? If you propose a fix, please make it concise.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

    Fix in Claude Code

Fix All in Claude Code

Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
backend/endpoints/sockets/scan.py:231-236
**Existing ROM identities stay empty**

When an existing library is upgraded and receives a default quick, scheduled, unmatched, or untargeted update scan, this predicate prevents `get_rom_files` from running. Because the migrations do not backfill the new columns, those ROMs continue returning null title IDs, save IDs, save usage, and file versions until a complete, hash, or explicitly targeted scan is run.

### Issue 2 of 2
backend/handler/filesystem/roms_handler.py:235
**Rename precedes database reconciliation**

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

Reviews (1): Last reviewed commit: "Merge upstream master into feature/sigil..." | Re-trigger Greptile

)
return None

os.rename(abs_file_path, new_path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Rename precedes database reconciliation

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

Knowledge Base Used:

Prompt To Fix With AI
This is a comment left during a code review.
Path: backend/handler/filesystem/roms_handler.py
Line: 235

Comment:
**Rename precedes database reconciliation**

Filename embedding commits the filesystem rename before the updated ROM and file records are persisted, with no rollback when a later scan step fails. This leaves the database pointing at the old path until recovery and can cause a watcher-triggered scan to register the renamed file separately.

**Knowledge Base Used:**
- [Filesystem Handler](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/filesystem-handler.md)
- [ROM Scanning Flow](https://app.greptile.com/romm/-/custom-context/knowledge-base/rommapp/romm/-/docs/rom-scanning-flow.md)

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves scan-time identification by extracting platform-native title IDs and save IDs directly from ROM binaries (via the optional sigil binding), persisting them in new DB columns, and surfacing them in the v2 UI. It also adds an opt-in Switch-only filename renaming mode to embed extracted title IDs and versions into filenames during scans, with DB reconciliation.

Changes:

  • Backend: Add title_id, save_id, save_usage, and per-file title_version fields, extract them during filesystem scanning, and persist them through scan reconciliation and file syncing.
  • Packaging: Build and install the optional sigil native bindings in Docker builds, and document optional local developer setup steps.
  • Frontend v2: Regenerate OpenAPI types and display per-file Title ID and Version on the Files tab, plus title/save IDs on the metadata tab.

Reviewed changes

Copilot reviewed 39 out of 44 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
frontend/src/v2/utils/romArtwork.test.ts Update test fixtures for new RomFileSchema fields.
frontend/src/v2/components/GameDetails/RelatedGameCard.vue Extend synthetic rom shape with new title/save fields.
frontend/src/v2/components/GameDetails/MetadataTab.vue Display ROM-level Title ID and Save ID in metadata rows.
frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue Display per-file Title ID and Version chips in file rows.
frontend/src/locales/en_US/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/en_GB/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/de_DE/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/es_ES/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/fr_FR/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/it_IT/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ja_JP/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ko_KR/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/pt_BR/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/pl_PL/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ru_RU/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/ro_RO/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/tr_TR/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/zh_CN/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/zh_TW/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/cs_CZ/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/bg_BG/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/locales/hu_HU/rom.json Add rom.save-id, rom.title-id, rom.version keys.
frontend/src/generated/models/SimpleRomSchema.ts Regenerate API model to include title/save fields and SaveUsage.
frontend/src/generated/models/DetailedRomSchema.ts Regenerate API model to include title/save fields and SaveUsage.
frontend/src/generated/models/RomFileSchema.ts Regenerate API model to include per-file title/save/version fields.
frontend/src/generated/models/SaveUsage.ts Add generated SaveUsage union type.
frontend/src/generated/index.ts Export generated SaveUsage type.
Dockerfile Build and install sigil bindings into the runtime venv during image build.
docker/Dockerfile Add sigil-build stage and copy bindings into the slim image venv.
DEVELOPER_SETUP.md Document optional local build/install steps for sigil.
backend/tests/handler/test_fastapi.py Add tests ensuring extracted rom-level fields fold into Rom and are preserved on rescans.
backend/tests/handler/filesystem/test_roms_handler.py Add filesystem handler tests for extraction, categorization, base-ID derivation, and embedding renames.
backend/tests/endpoints/sockets/test_scan.py Add tests for rename reconciliation and for persisting scanned per-file sigil columns.
backend/tests/adapters/services/test_sigil.py Add unit tests for guarded import behavior and extraction field mapping.
backend/models/rom.py Add SaveUsage enum, new columns, and title-id indexes to Rom and RomFile.
backend/handler/scan_handler.py Persist extracted rom-level title/save fields, without wiping on extraction-disabled rescans.
backend/handler/filesystem/roms_handler.py Extract title/save IDs and per-file versions via SigilService, map Switch CNMT content types, and optionally rename Switch files.
backend/handler/database/roms_handler.py Persist new per-file scanned columns (title_id, save_id, title_version) during sync_rom_files.
backend/endpoints/sockets/scan.py Wire config-gated extraction and embedding into _identify_rom, including optional Switch prod.keys resolution.
backend/endpoints/responses/rom.py Expose new title/save fields (rom and file) plus SaveUsage in response schemas.
backend/config/config_manager.py Add config keys for skipping extraction and enabling Switch filename embedding.
backend/alembic/versions/0104_sigil_title_ids.py Add DB columns and indexes for rom/file title and save IDs, plus save_usage enum.
backend/alembic/versions/0105_sigil_title_version.py Add rom_files.title_version BigInteger column.
backend/adapters/services/sigil.py Add guarded sigil adapter service with platform gating, error handling, and field mapping.
Files not reviewed (5)
  • frontend/src/generated/index.ts: Generated file
  • frontend/src/generated/models/DetailedRomSchema.ts: Generated file
  • frontend/src/generated/models/RomFileSchema.ts: Generated file
  • frontend/src/generated/models/SaveUsage.ts: Generated file
  • frontend/src/generated/models/SimpleRomSchema.ts: Generated file
Comments suppressed due to low confidence (1)

frontend/src/v2/components/GameDetails/FilesTab/FileRow.vue:181

  • v-if="file.title_version" will not render the version chip when the version is 0, even though 0 is a valid extracted title version. Use a null/undefined check instead of truthiness.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 96 to 100
const hasAnyHash = computed(
() =>
Boolean(props.file.title_id) ||
Boolean(props.file.title_version) ||
Boolean(props.file.sha1_hash) ||
tmgast added 2 commits July 29, 2026 12:46
The 3DS nests its save under two directory levels, so sigil reports it as
a '/'-separated save_id (00040000/00033500) flagged with usage=folder-split
rather than a flat id. Add SaveUsage.FOLDER_SPLIT, migrate the saveusage
enum (0106), regenerate the frontend SaveUsage type, and bump the sigil
pin to 98f3c62.
@tmgast
tmgast force-pushed the feature/sigil-title-ids branch from 316611a to 0c88dfd Compare July 30, 2026 06:27
Align cleanup_orphaned_resources tests with upstream 06cafd4, which made
the task schedule unconditionally: assert the default cron is set, and
clear cron_string in the unschedule-path test.
@gantoine gantoine changed the title Add binary title ID and save ID extraction during scan feat: add binary title ID and save ID extraction during scan Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

on-hold Pending further research or blocked by another issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Additional Info on Switch TitleIDs

3 participants